Aggregations cancellation after collection #120944
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
not-napoleon
added
>bug
:Analytics/Aggregations
|
Hi @not-napoleon, I've created a changelog YAML for you. |
|
@not-napoleon I think we should also update the |
…cancellation-after-collection' into aggregations-cancellation-after-collection
…cancellation-after-collection' into aggregations-cancellation-after-collection
|
I have performance data! These come from esbench and it's been a while since I did a deep dive into how we get all these, but it sure looks like the overhead is seems to be in the ballpark of a couple of % for this super deep agg - the worst case. I'm rerunning rally against to get the results for all aggs. I expect we'll only see a performance hit on the super deep aggs. |
|
Here's the rally track, a worst case scenario for this fix: elastic/rally-tracks#733 |
|
Here's some more timing information. This one the "after" came in faster. Which makes me think i got a machine with quieter neighbors. |
|
In that last test indexing time got faster with our change the doesn't touch indexing: It's my guess that the first run had a more noisy neighbor. It's possible that something is different in That faster-ness seems to be reflected in some runs. Non aggs: And aggs: But some aggs aren't faster: Our agg did get faster: I'm going to say "noise". This gives me the fuzzy feeling that we're ok merging this and watching for a regression in the nightlies. It'll probably take a bit to see. |
nik9000
reviewed
Jan 29, 2025
| int maxBucket, | ||
| boolean isInSortOrderExecutionRequired, | ||
| MappedFieldType... fieldTypes | ||
| ); |
There was a problem hiding this comment.
That's a ton of stuff to pass in. I feel like some of it, like CircuitBreakerService should be part of the implementation. Maybe?
There was a problem hiding this comment.
I think it's fine actually - it lines up with how everything else is working now.
…cancellation-after-collection' into aggregations-cancellation-after-collection
…cancellation-after-collection' into aggregations-cancellation-after-collection
iverase
reviewed
Feb 3, 2025
| for (long owningBucketOrd = 0; owningBucketOrd <= oldOrds.maxOwningBucketOrd(); owningBucketOrd++) { | ||
| long maxOwning = oldOrds.maxOwningBucketOrd(); | ||
| for (long owningBucketOrd = 0; owningBucketOrd <= maxOwning; owningBucketOrd++) { | ||
| if (context.isCancelled()) { |
There was a problem hiding this comment.
Nit: It is not clear to me why we are handling auto-date histogram in a special way. Could we add a comment explaining the why?
There was a problem hiding this comment.
Sure! Its just that it spends a while spinning around doing extra stuff before it's ready for the normal build.
There was a problem hiding this comment.
And I found it from an actual bug report of folks that were trying to terminate aggs and the thread was stuck here.
There was a problem hiding this comment.
you have not one but three typos in that sentence :D
iverase
approved these changes
Feb 3, 2025
…cancellation-after-collection' into aggregations-cancellation-after-collection
not-napoleon
added a commit
to not-napoleon/elasticsearch
that referenced
this pull request
Feb 6, 2025
This PR addresses issues around aggregations cancellation, mentioned in elastic#108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree. Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to. --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Nik Everett <[email protected]>
💔 Backport failed
You can use sqren/backport to manually backport by running |
elasticsearchmachine
pushed a commit
that referenced
this pull request
Feb 6, 2025
This PR addresses issues around aggregations cancellation, mentioned in #108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree. Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to. --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Nik Everett <[email protected]>
not-napoleon
added a commit
to not-napoleon/elasticsearch
that referenced
this pull request
Feb 6, 2025
This PR addresses issues around aggregations cancellation, mentioned in elastic#108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree. Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to. --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Nik Everett <[email protected]> Conflicts: server/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java
elasticsearchmachine
pushed a commit
that referenced
this pull request
Feb 6, 2025
This PR addresses issues around aggregations cancellation, mentioned in #108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree. Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to. --------- Conflicts: server/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java Co-authored-by: elasticsearchmachine <[email protected]>
not-napoleon
added a commit
to not-napoleon/elasticsearch
that referenced
this pull request
Feb 6, 2025
This PR addresses issues around aggregations cancellation, mentioned in elastic#108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree. Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to. --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Nik Everett <[email protected]> Conflicts: test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java
not-napoleon
added a commit
to not-napoleon/elasticsearch
that referenced
this pull request
Feb 6, 2025
This PR addresses issues around aggregations cancellation, mentioned in elastic#108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree. Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to. --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Nik Everett <[email protected]> Conflicts: test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java
elasticsearchmachine
pushed a commit
that referenced
this pull request
Feb 6, 2025
This PR addresses issues around aggregations cancellation, mentioned in #108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree. Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to. --------- Conflicts: test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java Co-authored-by: elasticsearchmachine <[email protected]>
not-napoleon
added a commit
to not-napoleon/elasticsearch
that referenced
this pull request
Feb 6, 2025
This PR addresses issues around aggregations cancellation, mentioned in elastic#108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree. Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to. --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Nik Everett <[email protected]> Conflicts: test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java
elasticsearchmachine
pushed a commit
that referenced
this pull request
Feb 6, 2025
This PR addresses issues around aggregations cancellation, mentioned in #108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree. Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to. --------- Conflicts: test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java Co-authored-by: elasticsearchmachine <[email protected]>
not-napoleon
added a commit
that referenced
this pull request
Feb 7, 2025
This PR addresses issues around aggregations cancellation, mentioned in #108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree. Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to. --------- Conflicts: test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java Co-authored-by: elasticsearchmachine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses issues around aggregations cancellation, mentioned in #108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree.
Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to.
I'm opening this PR as a draft while we work on testing for this change, to have a place for discussion and to add testing. There are three major pieces of testing we need to do:
Nik and I will be adding tests to this PR over the next few days as we work on it.
Fixed #108701